Statistical Signal Processing
This page contains resources about Statistical Signal Processing, Point Estimation, Estimation Theory, Adaptive Filtering, Adaptive Signal Processing and Adaptive Filter Theory, and System Identification. Subfields and Concepts * Linear Shift-Invariant (LSI) filter * Adaptive filter ** Shift-Varying filter ** Adaptive Algorithm * Four major configurations of an adaptive filter ** System Identification ** Noise Cancellation ** Prediction ** Inverse System Identification * FIR filter / Non-recursive adaptive filter ** Gradient Descent filter / Steepest Descent filter ** Least Mean Squares (LMS) filter ** Normalized LMS (NLMS) filter ** Leaky LMS filter ** Gradient Adaptive Lattice (GAL) filter ** Joint Process estimator ** FIR Wiener filter * IIR filter / Recursive adaptive filter ** Recursive Least Squares (RLS) filter ** Exponentially Weighted RLS filter ** Sliding Window RLS filter ** QR Decomposition based RLS (QRD-RLS) ** IIR Wiener filter * Online Estimation / Recursive Estimation ** Recursive least squares (RLS) filter / Forgetting Factor ** Kernel RLS (KRLS) filter ** Kalman filter ** Extended Kalman filter (EKF) ** Unscented Kalman filter (UKF) * Kernel LMS (KLMS) filter * Hierarchical LMS (HLMS) filter * Complex valued filters ** Complex LMS (CLMS) filter * Square Root filter * Monte Carlo Methods ** Particle filter * Optimum filters ** Eigenfilter ** Kalman filter ** Wiener filter * Time domain methods * Frequency domain methods / Spectral Estimation ** Nonparametric Methods *** Periodogram *** Modified Periodogram *** Barlett's Method / Periodogram Averaging *** Welch's Method / Modified Periodogram Averaging *** Blackman-Tukey Method / Periodogram Smoothing *** Multiple Window Method *** Empirical Transfer Function Estimation (ETFE) ** Minimum Variance Spectral Estimation (MVSE) ** Maximum Entropy Method (MEM) ** Parametric Methods *** AR Spectral Estimation (using All-Pole Transfer Function) *** MA Spectral Estimation (using All-Zero Transfer Function) *** ARMA Spectral Estimation (using Pole-Zero Transfer Function) *** Minimum Variance Distortionless Response (MVDR) filter *** Linearly Constrained Minimum Variance (LCMV) filter *** Subspace Methods ** Frequency Estimation (using a Harmonic Process / Sinusoid Model) *** Pisarenko Harmonic Decomposition *** MUSIC Pseudospectrum *** Eigenvector Method *** Minimum Norm Algorithm *** Subspace Methods / Eigendecomposition-based Methods **** Blackman-Tukey Principal Components Method **** AR Method **** Minimum Variance Method ** High-resolution / Super-resolution Spectral Estimators *** MVDR *** LCMV *** MUSIC * Bayesian Linear Regression * Regularization ** Regularized least squares ** L1-regularization / LASSO ** L2-regularization / Ridge Regression * Iterative Methods ** Expectation-Maximization (EM) Algorithm ** Levenberg–Marquardt Algorithm ** Iteratively Reweighted Least Squares ** Nonlinear Least Squares ** Gradient Descent / Steepest Descent ** Krylov Subspace Methods *** Conjugate Gradient Method ** Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm * Blind adaptive filter ** Blind Deconvolution * Risk Function / Expected Loss (i.e. Expectation Value of Loss Function) ** Mean Square Error (MSE) * Estimators ** Biased Estimator ** (Asymptotically) Unbiased Estimator ** Inconsistent Estimator ** (Asymptotically) Consistent Estimator * Frequentist Parameter Estimation ** Frequentist Risk * Bayesian Parameter Estimation / Bayesian Point Estimation / Bayesian Methods ** Posterior Risk / Bayesian Risk ** Bayesian Density Estimation ** Bayes filter / Recursive Bayesian Estimation *** Kalman filter ** Nonparametric Empirical Bayes (NPEB) ** Parametric Empirical Bayes Point Estimation ** Exponential family ** Conjugate prior * Methods for finding estimators ** Method of Moments / Moment Matching ** Maximum Likelihood Estimation (MLE) ** Maximum a posteriori (MAP) ** Pseudolikelihood ** Bayes Estimator / Bayesian Decision Theory *** Conjugate prior *** Exponential family *** Utility Theory ** Minimum-variance mean-unbiased estimator (MVUE) ** Median-unbiased estimator * Methods for evaluating estimators ** Minimum mean squared error (MMSE) / Bayes least squared error (BLSE) ** Best linear unbiased estimator (BLUE) ** James–Stein estimator * M-estimators ** MLE ** Steepest Descent / Gradient Descent *** Stochastic Gradient Descent (SGD) *** Generalised Normalised Gradient Descent (GNGD) *** Hierarchical Gradient Descent (HGD) * Stochastic Optimization * Cramér–Rao bound / Cramér–Rao Inequality * Sufficiency and Variance Reduction Techniques (VRT) ** Factorization Theorem ** Minimal Sufficient Statistic ** Rao–Blackwell theorem *** Rao–Blackwellization *** Rao–Blackwell estimator ** Exponential family ** Conjugate prior family * Black-box and grey-box models * Parametric and nonparametric models * Time Series Models ** ARMA Models ** Volatility Models / Conditional Heteroscedastic Models / ARCH Models ** ARMA Metrics (e.g. Martin Distance) * State Space Models ** Kalman filter ** State Space Models with Regime Switching / Jump Markov Linear Systems ** Subspace Identification *** N4ASID *** MOESP *** CVA / CCA *** SSARX *** Frequency domain subspace identification *** Time domain subspace identification * Stochastic Systems ** Stochastic Processes ** Stochastic Calculus ** Stochastic Optimal Control ** Stochastic Time Series * Signal Modelling (mainly using All-Pole Transfer Funcion) ** Least Squares Method ** Padé Approximantion ** Prony's Method ** Levinson–Durbin Recursion ** Lattice filter ** Iterative Prefiltering ** Final Data Records / Finite Iinterval Modelling *** Autocorrelation Method *** Covariance Method *** Modified Covariance *** Burg Algorithm ** Stochastic Modelling *** Modified Yule-Walker Equation (MYWE) Method *** Least Squares MYWE Method *** MA Model using Spectral Factorization *** Durbin's Method * Signal Detection Theory * Filtering Problem * The Expected Loss Principle * Spectral Factorization ** Innovations Filter * Bayesian Information Theory ** Minimum Message Length (MML) ** Occam's Razor * Model Order Selection / Model Comparison ** Akaike Information Criterion (AIC) ** Bayesian Information Criterion (BIC) ** Minimum Description Lenght (MDL) ** Akaike Final Prediction Error (FPE) ** Parzen's Criterion AR Transfer Function (CAT) * Nonlinear System Identification ** Artificial Neural Networks *** Recurrent Neural Networks *** Stochastic Neural Networks ** NARMAX Models ** Volterra Series Models ** Block Structured Models * Subspace Methods / Subspace Learning / Dimensionality Reduction ** Supervised Learning *** Feature Selection ** Unsupervised Learning *** Principal Components Analysis (PCA) ** Subspace Tracking *** Grassmannian Rank-One Update Subspace Estimation (GROUSE) *** Parallel Estimation and Tracking by REcursive Least Squares (PETRELS) *** Multiscale Online Union of Subspaces Estimation (MOUSSE) *** Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA) *** Online Supervised Dimensionality Reduction (OSDR) * Adaptive Control ** Auto-tuning / Self-tuning Online Courses Video Lectures * Adaptive filters by Ali H. Sayed * Learning Theory by Reza Shadmehr Lecture Notes * Advanced Signal Processing by Danilo Mandic * Spectral Estimation and Adaptive Filtering by Danilo Mandic * Stochastic Systems by Florian Herzog * System Identification by Roy Smith * Recursive Estimation by Rafaello D'Andrea Books and Book Chapters * Candy, J. V. (2016). Bayesian signal processing: Classical, modern and particle filtering methods. 2nd Ed. John Wiley & Sons. * Box, G. E., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: forecasting and control. John Wiley & Sons. * Kumar, P. R., & Varaiya, P. (2015). Stochastic systems: Estimation, identification, and adaptive control. SIAM. * Haykin , S. (2014). Adaptive filter theory. 5th Ed. Prentice Hall. * Tangirala, A. K. (2014). Principles of System Identification: Theory and Practice. CRC Press. * Goodwin, G. C., & Sin, K. S. (2014). Adaptive filtering prediction and control. Dover Publications. * Åström, K. J., & Wittenmark, B. (2013). Adaptive control. Dover Publications. * Van Trees, H. L. (2013). Detection Estimation and Modulation Theory, Part I: Detection, Estimation, and Filtering Theory. 2nd Ed. John Wiley & Sons. * Aster, R. C., Borchers, B., & Thurber, C. (2012) Parameter estimation and inverse problems. Academic Press * Diniz, P. S. (2012). Adaptive filtering. Springer . * Sayed, A. H. (2011). Adaptive filters. John Wiley & Sons. * Adali, T., & Haykin, S. (2010). Adaptive signal processing: next generation solutions. John Wiley & Sons. * Hayes, M. H. (2009). Statistical digital signal processing and modeling. John Wiley & Sons. * Mandic, D. P., & Goh, V. S. L. (2009). Complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models (Vol. 59). John Wiley & Sons. * Kulkarni, V. G. (2009). Modeling and analysis of stochastic systems. 2nd Ed. CRC Press. * Porat, B. (2008). Digital processing of random signals: theory and methods. Prentice-Hall. * Vaseghi, S. V. (2008). Advanced digital signal processing and noise reduction. John Wiley & Sons. * Van den Bos, A. (2007). Parameter estimation for scientists and engineers. John Wiley & Sons. * Poularikas, A. D., & Ramadan, Z. M. (2006). Adaptive filtering primer with MATLAB. CRC Press. * Manolakis, D. G., Ingle, V. K., & Kogon, S. M. (2005). Statistical and adaptive signal processing: spectral estimation, signal modeling, adaptive filtering, and array processing. Norwood: Artech House. * Gray, R. M., & Davisson, L. D. (2004). An introduction to statistical signal processing. Cambridge University Press. * Lehmann, E. L., & Casella, G. (2003). Theory of point estimation. Springer. * Casella, G., & Berger, R. L. (2002). Statistical inference. Cengage Learning. * Cichocki, A., & Amari, S. I. (2002). Adaptive blind signal and image processing: learning algorithms and applications. John Wiley & Sons. * Nelles, O. (2001). Nonlinear system identification: from classical approaches to neural networks and fuzzy models. Springer Science & Business Media. * Moon, T. K. S., & Wynn, C. (2000). Mathematical methods and algorithms for signal processing. Pearson. * Lennart, L. (1999). System identification: theory for the user. PTR Prentice Hall, Upper Saddle River, NJ. * Bretthorst, G. L. (1998). Bayesian spectrum analysis and parameter estimation (Vol. 48). Springer Science & Business Media. * Van Overschee, P., & De Moor, B. L. (1996). Subspace identification for linear systems: Theory—Implementation—Applications. Springer. * Kay, S. M. (1993). Fundamentals of Statistical Signal Processing, Vol I: Estimation Theory. * Therrien, C. W. (1992). Discrete random signals and statistical signal processing. Prentice Hall PTR. * Scharf, L. L. (1991). Statistical signal processing (Vol. 98). Reading, MA: Addison-Wesley. * Kay, S. M. (1988). Modern spectral estimation. Pearson Education. Scholarly Articles * Vaidyanathan, P. P. (2007). The theory of linear prediction. Synthesis lectures on signal processing, 2''(1), 1-184. * De Cock, K., De Moor, B., & Leuven, K. U. (2003). Subspace identification methods.'' Control systems robotics and automation, Vol. 1 of 3, pp. 933-979 Software * System Identification Toolbox - MATLAB * Adaptive Filters (Digital Signal Processing Toolbox) - MATLAB See also * Optimization * Optimal Control * Robust Control * Probabilistic Graphical Models Other Resources * Subspace Tracking by Laura Balzano * Course notes on Classical and Bayesian Statistics * Filtering, State Estimation, and Other Forms of Signal Processing - Notebook Category:Signal Processing Category:Probability and Statistics Category:Control Theory Category:Machine Learning